One might expect overlay support for X would require an extension. In fact, no additional requests or events are needed to support overlays so a true X extension is unnecessary. The core X protocol's visual mechanism provides a way to create windows of different types. The only thing which is necessary is to support a way to advertise which visuals are overlay visuals since the core X notion of a visual does not include layer information.
By leaving a properly formatted property on the root window of each screen describing what visuals are for overlay planes, a client can inspect that property knowing its format and determine which visuals are overlay visuals. The included information would also indicate what layer the visual is in (remember, there can be multiple sets of overlays) and how transparency is implemented.
It is up to the client to select the visual appropriate to the client's needs. The standard CreateWindow protocol request is used to create a window. If the visual specified for the CreateWindow request is an overlay visual, the window will be created in the visual's specified layer. An Xlib programmer can simply call XCreateWindow with a Visual* of an overlay visual to create an overlay window.
So how does an X client determine what visuals are overlay visuals? The Overlay Visuals Convention specifies that a property named SERVER_OVERLAY_VISUALS should be placed on the root window of each screen supporting overlays by the X server. The X server itself creates the property. The property has a standard format. It consists of elements (as described in Figure 1) which specify a visual, the type of transparency supported by the visual, what layer the visual resides in, and a transparency value which can be treated as a mask or a pixel value depending on the type of transparency supported. The SERVER_OVERLAY_VISUALS root window property is expected to be of type SERVER_OVERLAY_VISUALS and must be 32-bit in format.
The transparency type is an enumerated value indicating how transparency works for the visual. The following transparency types are possible:
Figure 1: SERVER_OVERLAY_VISUALS property entry format.